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Gesture-speech based HMI for a rehabilitation robot 

Shoupu Chen; Kazi, Z.; Beitler, M.; Salganicoff. M.; Chester, D.; Foulds, R, 
Southeastcon '96. 'Bringing Together Education. Science and Technology'., Proceedings of the 
11-14 Apr 1996 
Page(s): 29-36 

Summary: One of the most challenging problems in rehabilitation robotics is the design of an e 
machine interface (HMI) allowing the user with a disability considerable freedom and flexibility, 
user direction approach combining comma 
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Toward multimodal human-computer Interface 

Sharma, R.; Pavlovic, V.I.; Huang, T.S. 
Proceedings of the IEEE 
Volume: 86 Issue: 5 May 1998 
Page(s): 853-869 

Summary: Recent advances in various signal processing technologies, coupled with an explos 
available computing power, have given rise to a number of novel human-computer interaction ( 
speech, vision-based gesture recognition, eye tra 

AbstractPlus | References | Full Text: PDF JEEE JNL 



Vi&w: 1-2 I View Search Resi 



Help Contact Us Privacy & S 



http://ieeexplore jeee.org/search/selected jsp?qry=%28%28gesture+%3Cand%3E+m 5/26/05 



DEEE Xplore - View Selected Items 



Page 1 of 2 




1^ View Selected Items 



Results for "((gesture <and> multimodal <and> speech )<in>metadata) 
Your search matched 65 of 1164322 documents. You selected 5 items. 



Wts^coma Uf^iied Stateis Paicfst and Tfademark Offsce 

eROWSE SEARCH l^.^.E XPLORE GUIDES 



Choose 
jCitation 
Download 

I EndNoteProCite,RefMan ipl| 



» le^rn more 



» Key 

IEEE JhiL IEEE Journal or 
Magazine 

{EE JUL lEE Journal or 
Magazine 



Display Forn^at: C Citation <S Citation & Abstract 
Article information 



IEEE Conference 
Proceeding 



lEECMF lEE Conference 
Proceeding 



IEEE 
STD 



IEEE Standard 



Viosv: 1-5 I Vie) 



Multimodal interfaces with voice and gesture input 

Milota, A.D.; Blattner, M.M. 

Systems, Man and Cybernetics, 1995, 'Intelligent Systems for the 21st Century'., IEEE Internet 
on 

Volume: 3 22-25 Oct 1995 
Page(s); 2760-2765 vol.3 

Summary: The modalities of speech and gesture have different strengths and weaknesses, bt 
create a synergy where each modality corrects the weaknesses of the other. The authors belie 
multimodal system such as one intertwining speech an 
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Word teaming in a multimodal environment 

Roy, D.; Pentland, A. 

Acoustics. Speech, and Signal Processing. 1998. ICASSP '98. Proceedings of the 1998 IEEE I 

Conference on 

Volume: 6 12-15 May 1998 

Page(s): 3761 -3764 vol.6 

Summary: We are creating human machine interfaces which let people communicate with ma( 
natural modalities including speech and gesture. A problem with current multimodal interfaces i 
forced to learn the set of words and gestures wh 
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Current trends in multimodal input recognition 

Salem, B.; Yates, R.; Saatchi, R. 

lEE Colloquium on Virtual Reality Personal Mobile and Practical Applications - 98/454 
28 Oct 1998 
Page(s): 3/1-3/6 

Summary: In order to optimise the effectiveness of a personal virtual reality (VR) system, it is r 
a natural and efficient way of interacting with it. This can be achieved by incorporation of a dire< 
user interface. To communicate 
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Toward multimodal interpretation in a natural speech/gesture interface 

Kettebekov, S.; Sharma, R. 

Information Intelligence and Systems, 1999. Proceedings. 1999 International Conference on 
1999 

Page(s): 328-335 

Summary: Hand gestures and speech comprise the most important modalities of human to hui 
Motivated by this, there has been a considerable interest in incorporating these modalities for "i 
computer interaction (HCI) part 
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Kettebekov, S.; Yeasin, M.; Sharma, R. 

Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society C 
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Volume: 1 1 8-20 June 2003 
Page(s): 1-565- 1-570 vol.1 

Summary: Despite recent advances in gesture recognition, reliance on the visual signal alone 
unrestricted continuous gesticulation is inherently error-prone. Since spontaneous gesticulation 
coverbal in nature, there have been some attem 
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L7: Entry 1 of 19 File: PGPB Apr 3, 2003 

PGPUB-DOCUMENT-NUMBER : 20030065505 
DOCUMENT-IDENTIFIER: US 20030065505 Al 

TITLE: Systems and methods for abstracting portions of information that is 
represented with finite-state devices 
PUBLICATION-DATE: April 3, 2 003 

US-CL-CURRENT : 704 /9 
INT-CL: [07] G06 F 17/27 

L7: Entry 2 of 19 File: PGPB Mar 6, 2003 

PGPUB-DOCUMENT-NUMBER: 20030046087 
DOCUMENT-IDENTIFIER: US 20030046087 Al 

TITLE: Systems and methods for classifying and representing gestural inputs 
PUBLICATION-DATE: March 6, 2003 

US-CL-CURRENT: 704 / 275 
INT-CL: [07] GIO L 21/00 



L7: Entry 3 of 19 File: PGPB Nov 1, 2001 

PGPUB-DOCUMENT-NUMBER: 20010037201 
DOCUMENT-IDENTIFIER: US 20010037201 Al 

TITLE: Speech recognition accuracy in a multimodal input system 
PUBLICATION-DATE: November 1, 2001 

US-CL-CURRENT: 704 / 258 
INT-CL: [07] GIO L 13/00 



L7: Entry 4 of 19 File: USPT Mar 15, 2005 

US-PAT-NO: 6868383 

DOCUMENT-IDENTIFIER: US 6868383 Bl 

TITLE: Systems and methods for extracting meaning from multimodal inputs using 
finite-state devices 

DATE-ISSUED: March 15, 2005 
US-CL-CURRENT: 704 / 254 ; 704 / 251 , 715 / 863 
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L7: Entry 5 of 19 File: USPT Nov 23, 2004 

US-PAT-NO: 6823308 

DOCUMENT-IDENTIFIER: US 6823308 B2 

TITLE: Speech recognition accuracy in a multimodal input system 
DATE-ISSUED: November 23, 2004 

US -CL- CURRENT: 704 / 256 ; 704/ 236 , 704/239, 704 /255 
INT-CL: [07] GIO L 15/14, GIO L 15/08, GIO L 15/28 

L7: Entry 6 of 19 File: USPT May 11, 2004 

US-PAT-NO: 6735566 

DOCUMENT-IDENTIFIER: US 6735566 Bl 

TITLE: Generating realistic facial animation from speech 
DATE-ISSUED: May 11, 2004 

US-CL-CURRENT: 704/256; 345 / 473 , 704/270 

INT-CL: [07] GIO L 15/11/ GIO L 21/00, G06 T 15^/20 

L7: Entry 7 of 19 File: USPT Apr 20, 2004 

US-PAT-NO: 6725199 

DOCUMENT-IDENTIFIER: US 6725199 82 

TITLE: Speech synthesis apparatus and selection method 
DATE-ISSUED: April 20, 2004 

US-CL-CURRENT: 704 / 258 ; 704 / 260 , 704 / 270. 1 
INT-CL: [07] GIO L 13/00 

L7: Entry 8 of 19 File: USPT Jan 20, 2004 

US-PAT-NO: 6681206 

DOCUMENT-IDENTIFIER: US 6681206 Bl 

TITLE: Method for generating morphemes 

DATE-ISSUED: January 20, 2004 
US-CL-CURRENT: 704 / 243 ; 704 / 245 , 704/257 

INT-CL: [07] GIO L 15/06 
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US-PAT-NO: 6665640 

DOCUMENT-IDENTIFIER: US 6665640 Bl 

TITLE; Interactive speech based learning/training system formulating search queries 
based on natural language parsing of recognized user queries 

DATE-ISSUED: December 16, 2003 

US-CL-CURRENT: 704/ 257 ; 704 / 270. 1 , 704 / 275 , 707/4 



INT-CL: [07] GIO L 15/18, GIO L 15/22, G06 F 17/30 



L7: Entry 10 of 19 File: USPT Jul 14, 1998 

US-PAT-NO: 5781179 

DOCUMENT-IDENTIFIER: US 5781179 A 

TITLE: Multimodal information inputting method and apparatus for embodying the same 

DATE-ISSUED: July 14, 1998 
US-CL-CURRENT: 345/ 157 ; 704/251 



INT-CL: [06] GO 9 G 5/08 



L7: Entry 11 of 19 File: DWPI Mar 15, 2005 

DERWENT-ACC-NO: 2005-281798 
ABSTRACTED-PUB-NO: US 68 68383B 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Multimodal recognition system for speech input to personal digital 
assistant, generates recognition model for other mode recognition systems with 
recognition results from one mode recognition system 

INT-CL (IPC) : GIO L 15/08, GIO L 15/24 

Derwent-CL (DC) : P86, TOl , T04 , W04 

EPI Codes: T01-C08A; T01-J10B2; T04-D07D; W04-V04A; 



L7: Entry 12 of 19 File: DWPI Jan 20, 2004 

DERWENT-ACC-NO : 2004-200431 
ABSTRACTED- PUB-NO: US 66812 0 6B 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Acoustic morphemes generating method for use in spoken dialog system, 
involves clustering salient phone-phrases into acoustic morphemes based on semantic 
and syntactic similarities 

INT-CL (IPC) : GIO L 15/06 

Derwent-CL (DC) : P86, TOl , U21 , WOl , W04 

EPI Codes: T01-F06; T01-J18; U21-C01E; W01-C02B9; W04-V01; 
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DERWENT-ACC-NO: 2004-058591 
ABSTRACTED- PUB-NO: US 6665640B 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Speech based interactive learning system for e-learning, processes 
recognized speech utterance data to form sentence data which is converted into 
speech query for identifying topic answer entry corresponding to speech query 

INT-CL (IPC) : G06 F 17/30, GIO L 15/18, GIO L 15/22 
Derwent-CL (DC) : P86, TOl , W04 

EPI Codes: T01-J30A; TOl-NOlDl; W04-V04A; W04-W05A; 



L7: Entry 14 of 19 File: DWPI Apr 3, 2003 

DERWENT-ACC-NO: 2003-708376 
ABSTRACTED- PUB-NO: US2 00300655 05A 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Abstraction method for finite-state device, involves abstracting certain 
aspects of gestural content and taking projection from resulting finite state 
machine after multimodal integration 

INT-CL (IPC) : GO 6 F 17/27 

Derwent-CL (DC) : TOl 

EPI Codes: TOl-JllAl; T01-J17; 



L7: Entry 15 of 19 File: DWPI Mar 6, 2003 

DERWENT-ACC-NO : 2003-439455 
ABSTRACTED- PUB-NO: US2 0030046087A 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Gesture representation method for wireless portable computing devices, 
involves providing recognized possible meaning lattice, based on gesture 
recognition lattice and recognized possible words sequences lattice 

INT-CL (IPC) : G^O L 21/00 

Derwent-CL (DC) : P86, TOl , T04 , W04 

EPI Codes: T01-J10B2; TOl-JllAl; T01-J18; T01-M06A1A; T04-D03; T04-D04; W04-V01; 
W04-V04A; W04-V05; 

L7: Entry 16 of 19 File: DWPI Apr 20, 2004 

DERWENT-ACC-NO : 2003-183511 
ABSTRACTED- PUB-NO: US20020184027A 
COPYRIGHT 2 005 DERWENT INFORMATION LTD 

TITLE: Speech synthesis apparatus for use as voice browser, assesses quality of 
speech form utterances to select speech synthesis engines with different 
characteristics 

INT-CL (IPC) : GIO L 13/00, GIO L 13/04 

Derwent-CL (DC) : P86, TOl , W04 

EPI Codes: T01-J18; T01-N03A1; W04-V04C; 
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L7: Entry 17 of 19 File: DWPI Nov 23, 2004 

DERWENT-ACC-NO: 2 002-218521 
ABSTRACTED- PUB-NO: EP 112 6436A 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Speech recognition method for complementary multimodal input system involves 
recognizing words by comparing identified features of speech and modal input data 
based on models for speech and event recognition 

INT-CL (IPC) : G06 F 3/16, GIQ L 13/00, GIO L 15/08, GIO L 15/14, GIO L 15/18, GIO L 

15/24 , GIO L 15/28 

Derwent-CL (DC) : P86, TOl , W04 

EPI Codes: T01-C08A; TOl-EOlC; T01-J05B4P; T01-J18; W04-V01; W04-V04C; 



L7: Entry 18 of 19 File: DWPI Mar 30, 2005 

DERWENT-ACC-NO : 2000-259225 
ABSTRACTED- PUB-NO: EP 992 933A 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Generating realistic facial animation directly from speech, analyses voice 
with regard to learned categories of facial gesture and trains hidden Markov models 
to obtain optimal representation for new vocal actions 

INT-CL (IPC) : GO 6 K 1/00, GO 6 K 1/68, GO 6 T 13/00, G06 T 15/70, GIO L 15/00, GIO L 
15/14, GIO L 21/00 
Derwent-CL (DC) : P86, TOl 

EPI Codes: T01-C08A; T01-J10C4; T01-J10C5; TOl- JIB; 



L7: Entry 19 of 19 File: DWPI Mar 28, 1997 

DERWENT-ACC-NO: 1997-250041 
ABSTRACTED- PUB-NO: JP 0908136 4A 
COPYRIGHT 2005 DERWENT INFORMATION LTD 

TITLE: Multimodal information input method for electronic computer - involves 
directing object displayed on monitor of computer using voice command converted 
into internal computer command by application program 

INT-CL (IPC) : G06 F 3/033, G06 F 3/16, G09 G 5/08^, GIO L 3/_00 
Derwent-CL (DC) : P85, P8.6 , TOl , W04 
EPI Codes: TOI-CIO; T01-J18; W04-V04; 
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